Setting Up AI Cluster
Create a Lossless Fabric for the AI workloads
After installing Cisco NDFC, you decide to test NDFC automation capabilities by deploying BGP EVPN fabric if the company decides to expand AI workloads across several sites in the future.
In this task, you will deploy the EBGP fabric with configured AI/ML QoS and queuing policies. This way, you will achieve lossless behavior in the network, which is required for the AI workload communication over RoCEv2. You will use Cisco NDFC templates to automate the deployment of fabric and QoS configuration.
Step 1
Step 2
Enter admin and C!sco123 in the username and password fields, respectively. Click the Login button.
Answer
You will see the Welcome, admin page with the Journey tab selected. The Getting Started Map is partially complete and it shows the next step, which is adding a fabric.
If you see an alert regarding privacy concerns, click Advanced, followed by clicking Proceed to 10.0.0.70 (unsafe), as depicted in the figures.
Step 7
Click the BGP Fabric template to highlight it, and then click the Select button.
Step 8
In the General Parameters tab, enter 65000 in the BGP ASN for Spines field. Verify that Multi-AS is chosen from the BGP AS Mode drop-down list, and check the Allow Same ASN On Leafs option.
Answer
Even though the Multi-AS setting allows you to use a unique ASN for each leaf, you also checked the option Allow Same ASN On Leafs. This option lets you configure the same BGP ASN on leafs. The same ASN on leaf switches is typically used when you want to create a single virtual port channel (vPC) pair.
Step 11
Click the Advanced tab. Scroll down and choose Enable from the Greenfield Cleanup Option drop-down list.
Step 12
Scroll further down and check the Enable AI/ML QoS and Queuing Policies check box. Choose AI_Fabric_QOS_25G from the AI/ML QoS & Queuing Policy drop-down list.
Answer
The AI_Fabric_QOS_25G template provides the QoS configuration that will be applied to the entire fabric, including PFC and ECN configuration. The PFC and ECN congestion mechanisms need to be enabled in tandem to ensure lossless communication across the fabric. You will inspect the QoS configuration in closer detail when you deploy the configuration to the switches.
Note
In this scenario, you selected the 25G policy, but typically you would select the template that reflects the speed of the physical ports on your switches.
Step 16
Back in your browser, scroll to the bottom, and paste the freeform configuration you have in the clipboard to the Intra-fabric Links Additional Config.
Answer
The freeform configuration window allows you to specify the commands that will be applied to intra-fabric links in addition to the configuration provided by the NDFC templates. These commands will enable PFC and QoS traffic classification on the interfaces between spine and leaf switches. The priority-flow-control watch-dog-interval on command enables the watch-dog timer that will monitor PFC frames and prevent potential PFC storms in the fabric.
Add Switches to the Fabric
Impressed with how easy it is to configure the BGP fabric with Cisco NDFC fabric templates, you decide to add your Cisco Nexus switches to the BGP fabric. But before pushing the configuration to the switches, you want to verify what kind of QoS configuration is NDFC deploying.
Step 19
Use the ssh admin@172.16.0.10 command to connect to the leaf01 switch. If prompted to continue connecting, enter yes and use C!sco123 as the password.
Step 20
Use the show run ipqos command to check if the switch has the QoS configuration applied to it.
Answer
You should see empty output. You can verify other switches using the same method. At this point, all switches should have no QoS configuration applied to it. In the following steps, you will use Cisco NDFC to apply the configuration by simply adding switches to the AI fabric that you created in the previous task.
Step 23
Enter the IP range 172.16.0.10-20 in the Seed IP field, admin as the username, and C!sco123 as the password. Verify that the Preserve Config check box is unchecked.
Answer
The provided IP range should discover both leafs and spine. You can always access their information in the lab job aid at the beginning of the lab guide. Cisco NDFC will automatically discover all devices within the seed range and devices that are within the defined Max Hops range from the seed switches. The Preserve Config check box allows you to add the switch to the fabric without losing any configuration. When the box is not checked, the switch configuration is cleared during the import process, which removes all configuration except the management details and reboots the switch.
Step 25
Click Confirm in the warning alert to proceed.
Answer
After a few seconds, you should see the discovered devices in the Discovery Results table. Specifically, you should see leaf01, leaf02 and spine01 discovered and their status reported as Manageable. Notice that NDFC pings all IP addresses within the provided range and reports the details about the device such as Serial Number and Model.
Step 26
Click the check box next to the Switch Name to choose the leaf01, leaf02, and spine01 switches. Then click the Add Switches button in the lower-right corner.
Answer
In the process of importing switches, Cisco NDFC erases the configuration on the switches and reboots it. The import process can take up to 10 minutes with physical switches, but with Cisco Nexus 9300v virtual switches the process will be finished within a few minutes.
Once the import process is complete, the switch Status changes to Switch Added, and the Progress bar changes to Green.
Step 28
Navigate to the Switches tab, and verify that all three switches are now added to the fabric.
Answer
If the Discovery Status column reports Unreachable, choose the unreachable switches, click the Actions button, select Discovery, and click the Rediscover button. By doing rediscovery, Cisco NDFC attempts to reconnect to the switch and obtain switch details. The Discovery Status on all three switches should be OK.
Note
The Operational Status column might indicate Minor alert status because some interfaces are initialized as down. This status has no effect on the lab task.
Step 29
Choose the spine01 switch, click the Actions button, and choose Set Role from the drop-down list.
Step 30
In the Select Role pop-up window, click Spine to highlight the choice. Then click the Select button in the lower-right corner of the pop-up window.
Step 31
Click the upper, dark blue Actions button that is located beside the Refresh icon, and choose Recalculate and Deploy from the drop-down list.
Answer
The Recalculate and Deploy action will start calculating the configuration that will be applied to the devices.
After the window with calculation progress closes, you should see the Deploy Configuration window.
The Fabric Status column will report the Out-of-Sync status, indicating you have pending configuration that can be deployed to the added devices. The Pending Config column shows how many lines of the configuration are pending to be deployed.
Note
The order of the devices that you see in the table might be different compared to the lab guide.
Step 32
Click the blue link under the Pending Config column in the leaf01 row to check which commands will be applied to leaf01.
Answer
After scrolling down, you should see the QoS configuration that will be applied to leaf01. This configuration is provided by the AI/ML QoS template that you included in the BGP Fabric template.
In the configuration below, the class-map type qos command ensures that all traffic marked with DSCP 26 will be classified as RoCEv2 traffic. Also, all traffic marked with a DSCP value of 48 will be marked as CNP (Congestion Notification Packet). The goal is to classify the RoCEv2 and CNP traffic, so that switches can put it in the right queue.
class-map type qos match-any ROCEv2 match dscp 26 class-map type qos match-any CNP match dscp 48
The policy-map configuration classifies appropriate traffic to the right queue. In this example, RoCEv2 traffic is classified to queue 3, and CNP control traffic is classified to the strict-priority queue 7 to deliver congestion notification packets as quickly as possible.
policy-map type qos QOS_CLASSIFICATION
class ROCEv2
set qos-group 3
class CNP
set qos-group 7
class class-default
set qos-group 0After traffic classification, you should see the queuing and scheduling configuration. The queue 3 (c-out-8q-q3) is assigned 50% of the bandwidth. The minimum threshold is set to 100 KB, the maximum threshold to 700 KB, and if there is congestion happening, mark the packets with ECN. The queue 7 (c-out-8q-q7) has priority 1 assigned to it. This queue handles the CNP packets.
policy-map type queuing QOS_EGRESS_PORT
class type queuing c-out-8q-q6
bandwidth remaining percent 0
class type queuing c-out-8q-q5
bandwidth remaining percent 0
class type queuing c-out-8q-q4
bandwidth remaining percent 0
class type queuing c-out-8q-q3
bandwidth remaining percent 50
random-detect minimum-threshold 100 kbytes maximum-threshold 700 kbytes drop-probability 40 weight 0 ecn
class type queuing c-out-8q-q2
bandwidth remaining percent 0
class type queuing c-out-8q-q1
bandwidth remaining percent 0
class type queuing c-out-8q-q-default
bandwidth remaining percent 50
class type queuing c-out-8q-q7
priority level 1To enable the PFC pause frames for a specific queue, an additional QoS configuration should be applied. In this example, the PFC pause frames are enabled on queue 3 and will be generated if congestion happens.
policy-map type network-qos qos_network
class type network-qos c-8q-nq3
pause pfc-cos 3
mtu 4200The service-policy commands are required to apply queuing and QoS configurations systemwide. So, if there is congestion, packets will be marked with ECN. The ports with PFC will receive those frames and send pause frames if congestion thresholds are triggered.
system qos service-policy type network-qos qos_network service-policy type queuing output QOS_EGRESS_PORT
Scroll further down to verify the configuration of interface Ethernet1/1. To implement end-to-end lossless fabric, PFC must be enabled on all interfaces across the network. The RoCEv2 traffic classification is enabled at the interface level, including the logical Network Virtual Interface (nve1) where traffic encapsulation and de-encapsulation occurs.
interface ethernet1/1 priority-flow-control mode on priority-flow-control watch-dog-interval on service-policy type qos input QOS_CLASSIFICATION no switchport ip address 10.4.0.2/30 description connected-to-spine01-Ethernet1/1 mtu 9216 ip pim sparse-mode no shutdown interface nve1 source-interface loopback1 host-reachability protocol bgp service-policy type qos input QOS_CLASSIFICATION
Step 34
Click the Deploy All button in the bottom-right corner of the Deploy Configuration - AI-Fabric window.
Step 38
Under Show Command 1, enter the run ipqos command, and click Execute in the bottom-right corner to verify if the QoS configuration was applied to leaf01.
Configure the Overlay Network
After deploying the BGP fabric on the switches and inspecting the AI/ML QoS policy configuration, you are gaining confidence in the automation capabilities of Cisco NDFC. You decide to also configure the EBGP overlay network to distribute reachability information about the endpoints. You will continue to use Cisco NDFC policies to configure leaf BGP ASNs and add your spine switches as EBGP neighbors to your leaf switches and the other way round.
Step 46
Under the Policies tab, click the Filter by attributes bar to open a drop-down filter list, and select Template.
Step 47
Choose contains from the drop-down list, and then type asn to complete the filter. Press Enter to search.
Answer
The filter field provides suggestions when typing and supports the autocomplete command when pressing the Tab key on the keyboard.
After performing the search, among the filtered policies, you will find the leaf_bgp_asn and bgp_asn policies that are created for each leaf switch. Notice that even though you configured only the leaf_bgp_asn policy, an additional bgp_asn policy was created for both leafs.
To configure the policies for EBGP neighboring between the leafs and spine, first you need to find out the IP addresses of the Loopback0 interfaces. The NDFC BGP template configured the loopback interfaces when you added switches to the fabric. You need to collect the IPs.
Step 50
Move the horizontal scroll bar to the right to see the IP/Prefix column and collect the IPs.
Answer
You should get similar outputs as shown below. In this example, the leaf01 Lo0 interface has IP address 10.2.0.1, the leaf02 Lo0 interface has IP address 10.2.0.3, and the spine01 Lo0 interface has IP address 10.2.0.2.
Note
Cisco NDFC assigns the Lo0 IP addresses when you add switches to the fabric, and your IPs might be different than shown in this example.
Step 55
In the Spine/Super Spine IP List field, enter the IP address that you have collected from the Lo0 interface on the spine01 switch. The BGP Update Source Interface by default should have the Loopback0 interface selected. Click Save in the bottom-right corner.
Answer
In this example, the IP is 10.2.0.2, but in your environment spine01 might have different IPs. Make sure to enter the IP that you have previously collected from the Interfaces tab.
After a couple of seconds, you should land back to the Policies tab and see a notification that the policies were successfully added.
Now you need to create a similar policy for the spine01 switch.
Step 60
In the Leaf IP List field, enter the IP addresses that you have collected from the Lo0 interfaces on the leaf01 and leaf02 switch. In the Leaf BGP ASN field, enter both leaf ASN numbers in a comma-separated list: 65010, 65010. The BGP Update Source Interface by default should have the Loopback0 interface selected. Click Save in the bottom-right corner.
Answer
In this example, leaf Lo0 IPs are 10.2.0.1 and 10.2.0.3, but in your environment, interfaces might have different IPs. Make sure to enter the Lo0 IPs that you have collected from the Interfaces tab.
After a couple of seconds, you should land back to the Policies tab and see a notification that the policies were successfully added.
After you configured the policies, you will need to deploy these changes to the switches. For any kind of configuration changes you make on fabric or policy level, Cisco NDFC must calculate the appropriate set of commands to reflect those changes.
Step 61
Click the blue Actions button in the top-right corner, and choose Recalculate and Deploy from the drop-down list.
Answer
You should see the recalculation progress in the temporary pop-up window.
After the calculation is done, you should be forwarded to the Deploy Configuration window. Because there are pending configuration changes on leaf01, leaf02 and spine01, Cisco NDFC is reporting the Fabric Status as Out-of-Sync on all three devices. If you want to inspect the pending BGP configuration, you can click the blue link under the Pending Config column for each device.
Configure the Network for AI Workloads
Having EBGP overlay configured and operational, you start thinking how to connect the AI cluster to the fabric. Because the AI cluster will generate a lot of traffic and process many sensitive documents, you decide to create a dedicated VRF and AI network subnet. This way, you will segregate the AI cluster from other workloads. You will also attach the AI network to the switch interfaces, so that endpoints Student VM and PC-01 get access to the network. Finally, by testing communication from Student VM (attached to Eth1/3 on leaf01) to PC-01 (attached to Eth1/3 on leaf02), you will test the end-to-end connectivity to confirm if the network is configured properly.
Note
The lab environment is using Cisco Nexus 9000v virtual switches. These switches allow you to configure QoS and congestion mechanisms for the AI network just like you would on the physical Cisco Nexus 9000 switches. Nonetheless, virtual Cisco Nexus 9000v switches cannot implement QoS policies and apply congestion mechanisms when congestion happens, that is only possible on physical Cisco Nexus switches. Therefore, within this lab exercise you will test the network connectivity for the configured network.
Step 64
In the Fabric Overview window, click the VRFs tab. Click the white Actions button and choose Create from the drop-down list.
Step 65
Change the VRF Name to AI_VRF_50000. Click the Propose VLAN button to allow Cisco NDFC to suggest the VLAN ID from the pool, and leave all other fields with the default values. Click Create in the bottom-right corner.
Answer
Because you are deploying this VRF for AI workloads, this task suggests the AI_VRF_50000 name, but you can choose any VRF name that you want.
After clicking Create, you will be redirected back to the VRFs tab, which is now showing the VRF that you have created. You should see the VRF status as N/A because it is not yet deployed on the switches.
Step 66
Click the Networks tab, then click the white Actions button and choose Create from the drop-down list.
Step 67
Change the Network Name to AI_Network_30000 and click the Propose VLAN ID button to get the available VLAN ID from the pool. Type 192.168.1.254/24 in the IPv4 Gateway/NetMask field and AI_Workloads in the VLAN Name field. To confirm network creation, click the Create button.
Answer
Because you are creating a dedicated network and VLAN ID for the AI workloads, this example suggests using the AI_ prefix for the network and VLAN name. You can choose other names, but it is good practice to define descriptive names that allow you to easily identify their purpose. Also notice the IP of the gateway that you defined—it is the default gateway your AI workloads will use to access the network.
After clicking Create, you will be redirected back to the Networks tab.
You should see your network for AI workloads in the Networks tab. The Network Status is reported as N/A because the network is not yet assigned to the interfaces and deployed.
Step 68
At the bottom of the screen, locate and click the terminal emulator icon, and type the ip a command to list the network interfaces of Student VM.
Step 70
Type ping 192.168.1.254, press Enter to check the reachability of the default gateway, and after 10 seconds press Ctrl+C to stop the pings. You will go back to the Cisco NDFC and leave the terminal window open.
Step 72
In the Fabric Overview window, click the Interfaces tab to see all physical and virtual interfaces on all the switches in the fabric that Cisco NDFC discovered.
Step 73
Click Filter by attributes. Create the Interface == Ethernet1/3 filter and press Enter to filter interfaces.
Answer
You should see three interfaces after applying the filter. The interfaces will be Down because they are not yet configured or attached to the network.
In the following steps, you will configure the Ethernet1/3 interfaces on leaf01 and leaf02 as access interfaces. You will attach them to the AI network, so that Student VM (connected to Ethernet1/3 on leaf01) and PC-01 (connected to Ethernet1/3 on leaf02) can communicate over the 192.168.1.0/24 AI network.
Step 75
Click the int_trunk_host string under the Policy field to change the policy template.
Answer
You should see the Select Attached Policy Template window.
By default, Cisco NDFC is using the int_trunk_host policy for this interface. You will reconfigure this interface as access interface by choosing a different interface template.
Note
Notice that the Edit Interfaces window shows you are editing 1 of 2 Selected Interfaces, specifically you are editing the leaf01 Ethernet1/3 interface. After you are done with editing this interface, you will repeat the process for the leaf02 Ethernet1/3 interface.
Step 76
Type access in the search bar, click the int_access_host policy template to select it, and click the Select button when you are done.
Step 78
Scroll further down to make sure that the Enable Interface, Enable Priority Flow Control, and Enable QoS Configuration check boxes are checked. Click the Save & Next button in the bottom-right corner.
Answer
The PFC and QoS configuration needs to be applied end-to-end to have lossless fabric. This includes configuring QoS and PFC on access interfaces as shown in this example. You have already configured the spine-to-leaf connections in the previous tasks when you configured the BGP fabric template.
After clicking the Save & Next button, you will repeat the same configuration steps, but this time for the Ethernet1/3 interface of the leaf02 switch.
Step 79
Click the int_trunk_host string under the Policy field to change the policy template.
Step 81
In the Access Vlan field, enter 2300. In the Interface Description field, type Link to PC-01 or something equally descriptive.
Answer
The configuration process for the Ethernet1/3 interface of the leaf02 switch will be very similar. Make sure that the Access Vlan value is the same ID value as you specified for your AI network VLAN ID and the leaf01 Ethernet1/3 interface. The description is now Link to PC-01, because it is the interface where PC-01 is connected.
Step 82
Scroll further down to make sure that the Enable Interface, Enable Priority Flow Control, and Enable QoS Configuration check boxes are checked. In the bottom-right corner, click the Save button first, and then click Deploy.
Answer
The configuration process for the Ethernet1/3 interface of the leaf02 switch will be very similar. Make sure that the Access Vlan value is the same ID value as you specified for your AI network VLAN ID. The description is now Link to PC-01, because it is the interface where PC-01 is connected.
After you click the Save button, you will get a notification in the bottom-right corner that you successfully edited the interfaces. Also, notice that the Deploy button is now clickable.
After clicking the Deploy button, you will be forwarded to the Deploy interfaces configuration screen showing you the Pending Configuration for the Ethernet1/3 interfaces.
Step 83
Click the Pending Config link for one of the interfaces to inspect the configuration.
Step 85
In the bottom-right corner, click the Deploy Config button to deploy the configuration.
Answer
After a few seconds, you should see the Deploy interfaces configuration window, showing the progress of the deployment.
After deployment is done, NDFC will automatically return you the Edit Interfaces window. In the bottom-right corner, you should notice a notification about successful configuration deployment.
Step 86
Click the X button in the top-right corner to exit the Edit Interfaces window.
Step 87
Choose again both Ethernet1/3 interfaces on leaf switches, click the white Actions button, and then choose Multi-Attach from the drop-down list.
Step 93
Use the ping 192.168.1.103 command to verify that Student VM can also reach PC-01, which is attached to the Ethernet1/3 interface on the leaf02 switch.
Answer
Notice that Student VM can also ping PC-01, which is connected to the other leaf switch.
With this task, you verified the connectivity between PC-01 and Student VM, which are communicating over a lossless network.
Note
The lab environment is using Cisco Nexus 9000v virtual switches. These switches allow you to configure QoS and congestion mechanisms for the AI network just like you would on the physical Cisco Nexus 9000 switches. Nonetheless, virtual Cisco Nexus 9000v switches cannot implement QoS policies and apply congestion mechanisms when congestion happens, that is only possible on physical Cisco Nexus switches. Therefore, this lab exercise does not cover congestion simulation.